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System and method for associating different types of media content 



The invention relates to a system for operating with different types of media 
content, the system being arranged to enable a user to use a first content of a first type. 

The invention also relates to a method of operating with different types of 
media content, the method comprising a step of identifying a user's usage of a first content of 
a first type. 



Document US2002/0002564A1 discloses data processing means for viewing, 
playback and delivery of continuous audiovisual data, media content such as motion pictures, 
video programs, audio programs, etc. The media content, e.g. MPEG-1 system stream or 
other format, and description data describing audio and video content are received by said 
means from outside, or said content and data can be stored in a database from which they can 
be retrieved. The description data are expressed through the use of, e.g. Extensible Markup 
Language (XML) whose standardization is pursued by the World Wide Web Consortium. 
The description data describe parameters of the audio and video content such as a title, 
caption, priority, location, start time, end time, etc. Video and audio streams of the media 
content are mutually synchronized and provided to video playback means and audio playback 
means. The media content is further rendered for the user. In the known data processing 
means, the user may request or disable simultaneous rendering of the provided audio and 
video contents. The user may choose to be provided with only the audio part or only the 
video part of the available media content. 

In the known data processing means, the audio and video contents correspond 
to each other because they are received together from a content provider, e.g. a TV 
broadcaster or the like. At the same time, the user may like simultaneous rendering of media 
contents which are not related by the content provider. It is a drawback of the prior art that 
user's preferences with respect to rendering the different contents are not taken into account. 
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It is an object of the present invention to provide a system for operating with 
different types of media content of the kind defined in the opening paragraph which does take 
into account the user's preferences with respect to simultaneous rendering of the different 
contents. 

5 The object of the invention is realized in that the system comprises identifying 

means for identifying that the user concurrently uses a second content of a second fype, said 
second content being unrelated with the first content, and associating means for associating 
said second content with the first content. 

The contents of different types, e.g. audio and video, which are not associated 

10 by any reference in any way in the system may be related in the user's mind. For example, 
the user may like simultaneous rendering of some contents of the different types, or the 
combination of the contents may carry the user back to some life experience, or the like. The 
system comprises identifying means and associating means for "registering" the relationship 
between such contents in the system. The identifying means identifies that the user uses, e.g. 

15 watches, listens to, etc, a first content of a first type and, possibly in a different way, a second 
content of a second type at the same time, wherein said first and second contents are not 
related in the system. The associating means associates said first and second contents to 
enable the system to exploit this relationship. 

One of the advantages of the present invention is that the user does not have to 

20 explicitly instruct the system to relate such contents because it may be done automatically 
and transparently for the user without hindering the user's usage of said contents. Another 
advantage is that the links between contents are made personal to the user of the system, 
which is not possible when the contents, e.g. a TV program or a movie which usually include 
audio and video, are obtained from the content provider. 

25 The object of the present invention is also realized in that the method of 

operating with different types of media content further comprises a step of identifying that 
the user concurrently uses a second content of a second type, said second content being 
unrelated with the first content, and a step of associating said second content with the first 
content. The method describes the functioning of the system of the present invention. 

30 



These and other aspects of the invention will be further elucidated and 
described with reference to the accompanying drawings, wherein: 

Fig. 1 illustrates the general scheme of the present invention; 
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Fig. 2 shows a functional block diagram of the system suitable for 
implementing the present invention; 

Fig. 3 shows an example of the associated contents of different types and the 
relationship between them; 

Fig. 4 shows an example of a time sequence of the user's usage of different 
media contents; 

Fig. 5 shows an embodiment of the method of the present invention. 
Throughout the Figures, the same reference numerals indicate identical or 
corresponding components. 



Many people today use a lot of media content. People watch, listen, browse 
different audio and video information, as well as create their own media content, e.g. by 
voice recording, taking pictures using a camera, making a home film, etc. Various consumer 
electronic devices 110 such as a television set (TV set) with a cable, satellite or other link, 
video cassette recorder (VCR), home stereo, internet broadcasts, world wide web, portable 
and stationary radio, CD player, personal digital assistant (PDA), etc. can be used for these 
purposes, as is shown in Fig. 1 . Each of these devices may have a user interface used to 
obtain, select, record, play, etc. the media content. Any one of the consumer electronic 
devices allows the user to use at least one media content, and some of these devices may 
output or input at least two contents of different types. The system according to the present 
invention comprises identifying means 120 for identifying that the user concurrently uses at 
least two contents of different types, whereas these contents are not unrelated in the system, 
e.g. the system does not include data indicating the relation between said contents. Such 
identifying means 120 may be any means capable of monitoring user's activities in the 
system, e.g. a monitoring camera connected to a dedicated data processing unit which is 
arranged to analyze the user's behavior by processing data obtained from the camera, said 
data processing unit may be a part of the consumer electronic device rendering the media 
content. 

One or many media devices capable of outputting and/or inputting the media 
content may be included in the system. When a plurality of such devices is included, they 
may have means (not shown) for communicating with each other, e.g. the devices may be 
connected into a network. The communication means may be arranged to obtain information 
about the user's usage of the content from each of the devices, and recognizing individual 
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devices providing said information. For example, each device outputs to the communication 
means its identification and information pertaining to the contents used by the user, e.g. the 
type of the used content(s) which may be one of predetermined types in the system, etc. 
Further details which concern functioning of the identifying means will be apparent to the 
5 person skilled in the art. 

The media content may basically comprise at least one or any combination of 
visual information, audio information, text, data describing the media content or its 
reproduction, or the like. The expression audio data, or audio content, is hereinafter used as 
data pertaining to audio comprising audible tones, silence, speech, music, tranquility, external 

10 noise or the like. The expression video data, or video content, is used as data which are 
visible such as a motion picture, static image, characters etc. The user device may be 
arranged to obtain the media content stored on different carriers such as, audio tapes, video 
tapes, optical storage discs, floppy and hard drive disks, etc. in any format, e.g. MPEG 
(Moving Picture Experts Group), MIDI (Musical Instrument Digital Interface), Shockwave, 

15 QuickTime, WAV (W aveform Audio), etc. The content may also be broadcast by a content 
provider, e.g. TV or radio, to be received by the user devices rendering said content for the 
user. The user may use the media content at home, public places, in public transport and so 
on. Examples of the different types of content may be a playlist of songs or music, moving 
pictures, still images or sets of the still images, e.g. an image slide, animation, text, meta-data 

20 related to the media content, etc. For instance, the meta-data may define the radio content, 
e.g. radio stations and the most frequently broadcast content that is available in the user's 
living area, the different kinds of playlists for setting up a collection of the media contents. 
The meta-data may, for example, include the music playlist, e.g. in which the next song is 
started when the current one ends, the radio station or television channel playlist, e.g. 

25 comprising a collection of favorite channels to watch or surf through. 

With reference to Fig. 2, a functional block diagram of the system of the 
present invention is shown. The system 200 comprises the identifying means 120 and 
associating means 210 for associating said two or more contents, simultaneous use of which 
is identified by the identifying means 120. The identifying means outputs to associating 

30 means the information concerning the contents concurrently used in the system. The 

associating means may be arranged to establish data which link said associated contents, such 
as meta-data comprising the information pertaining to a relationship between the associated 
contents. In one example, the meta-data may include information about the associated 
contents such as a name/title, source of or link to the contents, e.g. identification of the 
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device(s) used for rendering the content; observed user's behavior during the use of the 
content(s), e.g. types of the user's behavior like watching, listening, etc. may be 
predetermined, whether or not the user initiated the content rendering, date, start and/or end 
time of the content usage, duration of rendering, etc. as is shown in Table 1. Of course, some 
of the information included in the meta-data may be obtained from the identifying means 
120. 
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The associating means may be arranged to associate the contents automatically 
or to communicate information pertaining to the contents to the user for obtaining his/her 
approval for associating these contents. The system may also simply notify the user about 
associating the corresponding contents. 
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The system may optionally comprise storage means 220 arranged to store the 
meta-data, selection means 230 arranged to select the content, output means 240 arranged to 
output the associated contents, and rating means 250 arranged to rate the associated content 
The meta-data may be implemented in any standard data structure or database, 
5 any language, e.g. XML, and include any amount of information about the associated 

contents which is sufficient to distinguish them. In one example, the meta-data may be stored 
in the storage means 220 together with the corresponding contents, e.g. in a header of the 
corresponding media data. The new meta-data may be established whenever some contents 
are associated by the associating means. Thus, a plurality of meta-data records may be stored 
10 by the storage means. 

Given meta-data storing information about two associated contents, the 
selection means 230 may be arranged to identify the first content upon selection of the 
associated second content, and vice versa. Thus, the system can always find all associated 
contents if one of them is selected and the relationship between these associated contents has 
15 been established. The output means may be arranged to play, reproduce, etc. the associated 
contents simultaneously. 

The selection means may be further arranged to function as a recommender of 
the media content. The recommender may exploit the meta-data for recommending the 
content in accordance with the user's preferences. In that way, the recommendations can be 
20 personalized for the user. 

The recommender may be arranged to recommend the associated contents if 
the user selects one content and likes to be recommended with another content of different 
type in accordance with his/her preferences. The recommender may be further arranged to 
recommend contents correlating to one of the associated contents e.g. having a similar 
25 subject, topic, category, genre, etc. For example, if the song and clip are associated, the 

recommender may recommend, upon the selection of the clip, the associated song and other 
songs which are similar to the associated song. 

The recommender may be arranged to recommend the content on the basis of 
ratings provided by the rating means 250. The meta-data may be interpreted in different ways 
30 for rating them, e.g. as an explicit or implicit feedback from the user. For example, the media 
contents may be considered as explicitly associated by the user if the user has explicitly 
initiated reproduction, or the other way of a user's consumption, of both media contents, e.g. 
user's requests for rendering the contents were inputted to the system. The rating means 250 
may highly rate such association of the contents, or a high reliability may be assigned to such 
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an association. As shown in Table 1, the (N+l) meta-data record indicates that the user 
initiated displaying the video content C and playing the audio content D. Therefore, the 
contents C and D are associated with the maximum rating, for example with one point as a 
maximal possible rating. If one of the contents was not explicitly chosen for rendering or 
using it otherwise by the user, then the association of these contents may be rated to be lower 
or given less reliability. For example, in Table 1 the contents A and B are associated with a 
rating lower than the rating of contents C and D, namely with 0.5 point out of 1, because the 
content B, a song, was played on the radio, and the user did not initiate this. 

Other criteria for evaluating the association of the contents may be used by the 
rating means. For example, the association of die media contents may also be influenced by 
the user's behavior during simultaneous rendering of the content, a user's reaction for 
initiating simultaneous rendering of the contents when it was started by the system itself, and 
so on. Other parameters and characteristics of the associated contents, user's usage of the 
associated contents, etc. may be stored in the meta-data and used for their rating. 

In a further embodiment, the rating means may rate the associated contents 
depending on the number of times of using the associated contents simultaneously. For 
instance, if the contents were often used together, they are given a high rating. 

In a further embodiment, if the first content is associated with the second 
content and with a third content, the rating means may be arranged to rate said associations of 
the first content with the second content, and the first content with the third content 
differently. For example, the association of the first content and second content may be rated 
to be higher if the first content was more often used with the second content than with the 
third content. In another example, the association of the first content may be rated to be 
higher with the second content if the simultaneous usage of the first and second contents is 
more recent than the usage of the first and third contents. Thus, such associated contents are 
rated to be higher if they were recently used. 

An example of associated contents is shown with reference to Fig. 3. Content 
"audio 1" 310 is associated with content "video 1" 320. Such an association illustrates the 
idea of the present invention about association of the contents of different types. At the same 
time, content "video 1" 320 may be further associated with the other contents of different 
type, e.g. "audio 2" 330, "audio 3" 340, etc. The type of content "music playlist" 350, which 
is also associated with content "video 1" 320, may be considered as a separate type of the 
content corresponding to the group of contents of type "music". In turn, contents "audio 1" 
310 and "audio 3" 340 maybe associated with content "image 1" 360 and content "video 1" 
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320. In this example, if the user selects content "audio 3" and requests the recommender to 
recommend the media content of type different from type "audio", the recommender may 
recommend contents "image 1" 360 and 'Video 1" 320 because they are associated with the 
user-selected content "audio 3" 340. 
5 With reference to Fig. 4, an example of media contents used at different time 

in the system is shown. "Media content 1" 410 was used in the system from time Tl 1 (41 1) 
till time T12 (412), ''media content 2" 420 - from time T21 (421), "media content 3" 430 - 
from time T31 (43 1). Contents 410, 420 and 430 have different time lengths for their usage. 
The usage of content 410 was started earlier than the usage of content 420, though this 

10 difference in time (T21-T1 1) is much less than the duration (412-421) of the simultaneous 
usage of contents 410 and 420. The identifying means 120 may be arranged to provide a 
starting time, ending time, time of simultaneous usage of the contents to the associating 
means 210 for including said time information in the corresponding meta-data. The 
associating means may be arranged to estimate a ratio between the duration of simultaneous 

15 usage (412-421) of the corresponding contents and the time difference (T21-T1 1) or 
difference of times when the use of corresponding contents ends. In one example, the 
associating means may be arranged to ignore the simultaneous usage of the contents and the 
respective meta-data may not be established if the duration of simultaneous usage of them is 
much smaller than their usage non-overlapping in time. For example, the duration (412-43 1) 

20 of simultaneous usage of contents 41 0 and 430 is rather small with respect to the duration of 
their usage non-overlapping. The associating means may include a value of said estimated 
ratio in the respective meta-data. In another example, the rating means may rate the 
association of contents based on said ratio. For instance, the association of contents 410 and 
420 may be rated to be higher than the association of contents 410 and 430 in view of the 

25 considerations above. 

In one of the embodiments of the present invention, the selection means 230 
may enable the user to modify the meta-data in the system. Of course, an appropriate user 
interface including input means, e.g. a keyboard, a mouse, a touch-screen, speech recognition 
means, etc. may be provided, and the output means 240 may be arranged to present the meta- 

30 data to the user via, for instance, a display device, audio reproduction means etc. 

The identifying means, associating means, selection means and rating means 
may be implemented by using a microprocessor (not shown) coupled to a random access 
memory and a read-only memory (ROM) storing a program which, when executed by said 
microprocessor, can perform functions of said means as described above. The storage means 
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220 may be realized with said ROM. The media content may be stored locally in the system 
or downloaded from an external source by means of suitable arranged data receiving means 
(not shown). The realization of such a microprocessor system with corresponding internal 
and external circuits will be apparent to the person skilled in the art and need not be 
5 discussed herein. 

With reference to Fig. 5, the method of the present invention describing the 
operation of the system of the present invention is shown. According to the method of 
operating with different types of media content, the concurrent usage of the first content and 
second contents of different types is identified in step 510, wherein the relationship of said 

10 contents is not known in the system. In step 520, said first and second contents are associated 
as being related by their simultaneous user's usage. For instance, the user may watch a slide 
show of pictures on the display screen together with a certain song being played by the CD 
player in the system, or the user may often listen to some songs when the user would like to 
download certain pictures. Such contents being used by the user simultaneously are 

15 associated. Embodiments of the method, which correspond to the above-described 
embodiments of the system of the present invention, may be envisaged. 

The various program products may implement the functions of the system and 
method of the present invention and may be combined in several ways with the hardware or 
located in different devices. Variations and modifications of the described embodiment are 

20 possible within the scope of the inventive concept. Thus, for example, the use of the verb 'to 
comprise' and its conjugations does not exclude the presence of elements or steps other than 
those defined in a claim. The invention can be implemented by means of hardware 
comprising several distinct elements, and by means of a suitably programmed computer. In 
the device claim enumerating several means, several of these means can be embodied by one 

25 and the same item of hardware. 

A 'computer program' is to be understood to mean any software product 
stored on a computer-readable medium, such as a floppy-disk, downloadable via a network, 
such as the Internet, or marketable in any other manner. 



